It is often desired to display a large image or video in a different (usually smaller) size. This is common, for example, when generating image thumbnails, when obtaining short summaries of long videos, or when displaying images or videos on different screen sizes. It is generally desired that the smaller representation or the visual summary faithfully represent the original visual appearance and dynamics as best as possible, and be visually pleasing.
The simplest and most commonly used methods for generating smaller-sized visual displays are scaling and cropping. Image scaling maintains the entire global layout of the image, but compromises its visual resolution, and distorts the appearance of objects when the aspect ratio changes. Cropping, on the other hand, preserves visual resolution and appearance within the cropped region, but loses all visual information outside that region.
More sophisticated methods have been proposed for automatic “retargeting” by reorganizing the visual data (image or video) in a more compact way, while trying to preserve visual coherence of important (usually sparse) regions. These methods typically begin by first identifying important regions. The following articles describe some of these importance-based methods:
F. Liu and M. Gleicher. Automatic image retargeting with fisheye-view warping. In UIST. 2005.
V. Setlur. S. Takagi. R. Raskar. M. Gleicher. and B. Gooch. “Automatic image retargeting.” In MUM. 2005.
L. Wolf. M. Guttmann. and D. Cohen-Or. “Non-homogeneous content-driven video-retargeting.” In ICCV'07.
Existing retargeting methods can roughly be classified into three families:
(i) Importance-based scaling methods first identify important regions within the image (e.g., salient regions, faces, high-motion regions). The outputs of these methods are characterized by scaling-down of unimportant regions (e.g., the background), while the important regions are preserved as close as possible to their original size (e.g., foreground objects). These methods work well when there are only a few “important” objects within an image. However, these methods reduce to pure image scaling if there is uniform importance throughout the image.
(ii) Importance-based cropping methods provide acceptable results when the interesting information is concentrated in one region (spatial or temporal).
(iii) Object segmentation methods correct for the main deficiency of cropping—the inability to capture spatially or temporally separated objects—by compact packing (spatial and/or temporal) of segmented important/salient regions/blobs.
Most importance-based methods require the important regions to be relatively compact and sparse within the visual data. In contrast, the “Seam Carving” approach (described in the article “Seam carving for content-aware image resizing” by S. Avidan and A. Shamir, SIGGRAPH, 2007) does not rely on compactness/sparseness of important information. It removes uniform regions scattered throughout the image, by carving out vertical and horizontal pixel-wide seams that have low gradient content. As long as there are enough low-gradient pixels to remove, the results are pleasing. However, eventually all of the low gradient pixels have already been removed and further shrinking by “Seam Carving” can actually deform important image content. This is especially evident when the interesting object(s) span over the entire image.